Beta

AI Personalization is now available in public beta. Migrate now to personalize search results based on historical and predictive user affinities.

It’s best to A/B test your Personalization implementation before going live with it to all your users. This guide assumes knowledge of Algolia’s A/B testing feature and walks you through how to set up A/B tests for Personalization specifically.

Since you can save one Personalization strategy per application at a time, you can’t test different Personalization strategies against each other You can A/B test non-personalized search versus personalized search or different levels of Personalization impact.

If you’re already sending events and including the userToken parameter with your searches, you can A/B test Personalization entirely through the dashboard. For more information, see Create and run and A/B test.

​
Testing non-personalized versus personalized results

Once you’ve gathered user data and configured your Personalization strategy, it’s time to test personalized against non-personalized results.

Since you can enable Personalization at query time, you don’t need to create a replica index for this A/B test.

A/B testing Personalization isn’t a replacement for simulating Personalization while configuring your strategy. You should always simulate various users as part of your configuration process.

​
Using the dashboard

1

Turn off Personalization for the A variant

Ensure that Enable Personalization is off for the test index in the dashboard’s Indices section. You can find this and other settings under Configuration.

2

Set Personalization impact for the B variant

In the dashboard’s Personalization section, set the Personalization impact to a non-zero value. This ensures that when you enable Personalization for variant B, it has an effect.

3

Create an A/B test

Go to the dashboard’s A/B testing section. Click New test in the top left.

4

Name the test

Name your A/B test—something like “Non-personalized versus personalized results”.

5

Select an index for each variant

Name and select your variant indices. Name Variant A “Non-personalized” and Variant B “Personalized.” Select the same index for both.

6

Enable Personalization for variant B

Click + Add query parameter under Variant B. In the box that appears, select the Personalization tab. Set Enable Personalization and Tested value to on.

7

Set the percentage of traffic for variant B

Select the percentage of traffic to send to variant B (the personalized version). The more confident you are in your Personalization implementation, the higher value you can use. If you want to expose fewer users to Personalization, select a lower value. The more uneven the traffic split, the longer the test could take to reach statistical significance.

8

Determine test duration

Select the test duration. You should use at least two full business cycles. For example, if you see a predictable conversion trend over a week-long period, you should set the test to last at least two weeks. A shorter period might not include seasonality effects and end with biased results. Though not recommended, you can always stop a test early.

9

Save test

Click Create on the bottom right.

10

Verify test

From the A/B testing section, verify that you’ve set up your test as planned:

  • Both variants use the same index.
  • Variant B displays enablePersonalization:true underneath the index’s name.
  • The test is Running for the desired time.
  • The Traffic split between the variants is as planned.

If the setup isn’t correct, select Delete A/B Test and begin again.

Once the test is complete (and throughout the test if you wish), interpret the results.

​
Testing different Personalization impacts

Suppose you’ve already established that Personalization is creating a better experience for your users and helping you meet your business goals. In that case, you may want to fine-tune the Personalization impact—that is, the strength Personalization plays in your ranking formula.

For example, you can create an A/B test with different impacts:

  • Variant A with an impact of 50
  • Variant B with an impact = 70.

Don’t run mopre than one A/B tests on the same index at the same time. If you want to test both Personalization in general and different impact scores using the same index: complete one test before starting the next.